This invention relates to an imager, or camera, comprising at least one array of photodetector elements, here referred to as an image sensor, as well as imaging optics for projecting an image, such that at least part of the image reveals information about the spectral distribution of incoming light. More specifically, the invention relates to a compact device for multispectral or hyperspectral imaging which may also perform conventional imaging.
Hyperspectral imaging usually refers to imaging techniques which resolve the incoming light into more than 10 spectral bands, often hundreds of bands. Multispectral imaging usually refers to imaging with 2 to 10 bands. In both cases, the bands may be spectrally separate, adjacent or overlapping, although the term hyperspectral is usually understood as implying contiguous spectral bands. Here we will refer to multi- and hyperspectral imaging collectively as spectral imaging.
Although the different spectral components in the signals recorded by the imager are generally referred to as bands, they are more generally characterized by different spectral weightings of the incoming light. This spectral weighting may be effected in many ways, including filtering of the incoming light or adaptation of the different light sensing elements. Depending on the type of spectral imager, the recorded raw data may need to be processed in order to yield a final spectral image product, which we will refer to as the output image.
Spectral imaging records information present in the spectral distribution of incoming light in each image pixel. Most commonly, multispectral imaging is used for colour photography by sampling the light in 3 spectral bands representing the primary colours of the eye. It is well known that spectral information, such as colour in visual imaging, carries important information about objects in an image. The spectrum is generally a fingerprint of the composition of materials present within a pixel area in the imaged scene. For systems involving automated image analysis, the analysis may be facilitated, often to a significant degree, by availability of spectral information.
In many cases 3 spectral bands, such as in ordinary colour imaging, is suboptimal for image processing, and significant new information can be obtained from images with more spectral bands. Even a modest increase to about 6 to 10 spectral bands may provide significant new capabilities in many cases. In other cases it may be desirable to resolve the image spectrally into tens or hundreds of spectral bands. Typically, the technological complexity and cost increases with increasing number of bands. In a spectral imaging device, it is therefore desirable to choose the number of bands close to the minimum acceptable for the foreseen application.
In many practical applications, it is desirable to combine spectral imaging with conventional imaging modalities, such as colour or monochrome imaging where the output is two-dimensional images with limited or no spectral information. An example is the combination of spectral imaging with video for remote sensing target detection and identification. With technologies currently in use, combination of spectral imaging and conventional two-dimensional imaging has required systems containing separate imagers for these functions. This leads to a system with large size and weight, which would be an unacceptable burden on many sensor platforms of interest, such as unmanned aircraft or field portable equipment.
Any design of a spectral imager faces many conflicting requirements. Particular to spectral imaging is the need for spatial coregistration of different spectral bands: In any given pixel in the output image, all bands should sample the same pixel region in the scene, otherwise significant errors result [T. Skauli, Optics Express vol. 20 no. 2, p. 918-933]. Also, it is desirable to sample all spectral bands at the same time and angle, to avoid errors due to temporal or angular dependencies in the scene. Furthermore, since spectral imaging subdivides light into multiple spectral channels, it is desirable for an imager to collect a large amount of light and have a good optical throughput in order to have a good signal to noise ratio. In addition to these specific requirements for spectral imaging, any imager faces important constraints on size and cost.
Many technologies are in use for spectral imaging. Most colour imagers employ a single image sensor with an integrated array of colour filters, for example in a Bayer pattern [B. Bayer, U.S. Pat. No. 3,971,065]. The characteristics of the filter and photodetectors together define a set of different spectral responsivities, in this case adapted to match the different photoreceptors in the human eye. This works well for visual imagery. For imaging with a larger number of bands, however, the filter array concept becomes increasingly difficult to employ because of the large lateral separation of photodetector elements with different filters that contribute to a given pixel in the output image, leading to unacceptable coregistration performance. Also, a large fraction of the light is lost in each filter.
Some colour imagers, and other multispectral imagers, employ one photodetector array for each band, and use beamsplitters to direct one spectral band to each array. However this technology is limited in practice to at most about five spectral bands.
A simple multispectral imager can be built from a monochrome camera with wide spectral responsivity by placing it behind a spinning wheel consisting of multiple spectral filters. However the filter wheel concept suffers from loss of light in the filters as well as moving parts and non-simultaneous sampling of the spectral bands.
By imaging through a Michelson interferometer, it is possible to construct a spectral imager using the principle of Fourier transform spectroscopy. This technology is very favourable for its high optical throughput, but suffers from possible errors due to non-simultaneous sampling of spectral components, as well as large size and complications due to moving parts.
The most widely used technology for hyperspectral imaging is the imaging spectrometer. In this technology, a slit at the focal plane of an objective lens selects light from a region in the scene corresponding to one row of pixels in the output image. By a dispersive element (slit or grating) and reimaging optics, light from each pixel location along the slit is spectrally dispersed onto a column of photodetector elements in a detector array. Then the photodetector signals from one such column correspond to the spectrum of one image pixel. By scanning the field of view in a direction perpendicular to the slit, the sensor can image the two spatial dimensions. This technology offers good spatial and temporal coregistration of different bands, but suffers from low optical throughput due to the slit, as well as a relatively large size.
The most compact technology for hyperspectral imaging uses a bandpass filter in front of a photodetector array in the focal plane, fabricated in such a way that the passband wavelength varies across the image [A. M. Mika, “Linear-wedge spectrometer,” Proc. SPIE 1298, 127-131 (1990)]. The combination of filter characteristics and the spectral characteristics of the photodetector array defines a spectral responsivity which varies across the field of view. Typically the filter has a nearly constant spectral variation in one direction, known as a “linear variable filter” (LVF). By scanning the field of view in the direction of spectral variation, and repeatedly reading out the photodetector array, it is possible to assemble a hyperspectral image of an entire scene. Similarly, it is possible to perform multispectral imaging by placing a set of discrete spectral filters in the focal plane in such a way that a scanning motion enables recording of all bands from the entire scene.
This known concept for spectral imaging is illustrated in FIG. 1: A scene or object 1 is imaged by lens 2 onto a photodetector array 3 placed behind an “LVF” filter 4. The scene is scanned relative to the imager in a direction indicated by the arrow 5, leading to a corresponding movement of the image projected onto the photodetector array. For a given point in the scene, different spectral components are recorded at different positions of the scan movement. For this to yield a correct spectral image, the spectral signal from the scene must be independent of viewing angle changes and time during the scanning of any given point within the imaged area.
It can be seen that most of the spectral imaging technologies listed above, and most other technologies for spectral imaging, tend to be large and complex, hence also expensive and difficult to deploy in applications. The notable exceptions are the technologies based on filters in the focal plane. The linear variable filter imager still requires some form of scanning to image a scene. However, means for scanning is often provided in the application anyway, such as for an imager mounted on an airplane or on a pan/tilt actuator.
In U.S. Pat. No. 6,211,906B1, Sun describes a system in which two LVF-based imagers are combined with a conventional imager. This system uses the conventional imager to track the apparent motion of the scene, as seen from the imager. This tracking is used to aid the reconstruction of a hyperspectral image based on data from the two LVF-based imagers.
In International patent application WO2011/073430A1, Biesemans et al. describe an imaging system with two photodetector arrays. One of these arrays is used for conventional recording of two-dimensional frame images, monochrome or colour. The other array is covered by an LVF or by a multiband filter which passes different spectral bands to different regions of the array. The two-dimensional images are used for geometric positioning of the imagery from the filtered photodetector array, enabling the assembly of a spectral image. The authors note that the two-dimensional images may be used to reconstruct three-dimensional shape, as described previously in e.g. in an article by Marc Pollefeys and Luc van Gool “From images to 3D models”, pages 51-55, Communications of the ACM, July 202/vol. 45, No. 7.
It can be noted that for the filter-based spectral imaging techniques mentioned above, a precise geometric positioning is needed for each light sample read out from the array, as part of the construction of an output image. Also, this construction of the output spectral image must make the assumption that the scene is constant in time. Furthermore, since filters for different bands are located in different areas of the focal plane, different spectral bands will be viewed in different angles relative to the sensor. This is unproblematic in the case of a rotational scan around the principal point of the imaging optics. However if the scanning has a significant component of linear motion, such as for a sensor on an aircraft or a sensor viewing a conveyor belt, different bands will see the scene from different angles. Then the spectral signal from the scene must also be assumed to be independent of viewing angle. This assumption does not hold in all cases, and significant spectral errors may result for scenes which exhibit parallax effects or non-Lambertian BRDF.
A signal error due to viewing angle dependence may for example arise from parallax effects, as illustrated in FIG. 2. The figure shows an imager in two different positions C1 and C2, for example two points on the movement path of an aircraft, where the distance H is the imager height above ground. The scene has some 3D structure, represented in the figure by a building B with vertical walls and a height h. When the imager is in position C2, it observes a particular point P on the ground in a spectral band whose filter is positioned such that the line of sight is as indicated by the dashed line. This may, for example, be the shortest wavelength of an LVF. When the imager is in position C1, a different band is viewing in the direction of P, for example the longest wavelength of an LVF. However, the line of sight of this latter band is obscured by the roof of the building. Therefore, the amount of light observed in this band represents the roof material and not the ground material. It is thus not possible for the imager to obtain a spectrum from only the ground material at position P when following the path from C1 to C2. If, nonetheless, the recorded data are used to assemble a spectrum, which is then taken to represent position P, the resulting spectrum will have contributions from both the roof and the ground, and their relative contributions will vary between bands. In this case it is clear that large errors in the output image may result from the parallax effects in the scene.
In spectral imaging, it is normal to have some image pixels whose spectrum has contributions from different materials. Indeed, many algorithms for processing spectral images make the assumption that the scene is composed of several different materials which may be present in some mixture within a given image pixel. Crucially, for these algorithms to be applicable, the material mixture must be the same in all bands. If the material mixture differs between bands due to a shortcoming of the imaging process, for example as illustrated in FIG. 2, the spectral signal may be severely distorted. Such distortions are discussed in detail in [Skauli 2012].
If the spectrum from the scene exhibits an angle dependence due to parallax, the effect on the pixel spectrum will typically depend on the spatial resolution, or pixel size in the scene. In the particular case illustrated in FIG. 2, the amount of parallax distortion is expressed by the distance d. A significant error in the assembled spectrum results if this distance is comparable to or larger than the pixel size. Consider the case where the two lines of sight in FIG. 2 are separated by an angle a and represent the two spectral bands that are most widely separated in their viewing angle. Assume, for simplicity, that the imager moves in a straight path parallel to the direction of spectral variation of an LVF in the focal plane. Furthermore, assume that the photodetector array has square pixels so that the element spacing, or pixel pitch, is the same in the direction of movement and in the orthogonal direction. In the focal plane, the angle a corresponds to a distance between the photodetector elements used for recording the two bands. Let this distance be expressed by the number of pixel units between these photodetector elements, denoted S. Thus, S represents the maximum number of different spectral bands that can be recorded by the system. The size of image pixels on the ground is given byD=aH/S. 
The parallax distortion is given byd=ah. 
The parallax distortion can be expressed relative to the pixel size asE=d/D=Sh/H 
Thus for a scene which exhibits angle dependence due to parallax, represented by the example in FIG. 2, the ratio E must be well below 1 to ensure that the recorded spectra are not significantly distorted. Ideally, E should be comparable to, or less than, the relative contribution of noise to the spectral signal.
As an example, consider the highly realistic case of an airborne imager at an altitude H=1000 m and a building height of h=10 m. For an LVF-based spectral imager with S=100 bands, we find E=1. In other words, an LVF-based hyperspectral imager will easily exhibit large distortions in the spectra due to parallax effects, unless the viewing distance is large compared to the height of 3D structure in the scene, multiplied by the number of bands. In the case of a multispectral imager with S=10 bands, we find E=0.1 Even this will tend to result in unacceptable signal distortion, on the order of 10% of the signal. Furthermore, it may be difficult to fabricate an LVF or multiband filter with spectral characteristics that vary in a controlled manner over a very short distance.
Thus it is an object of the present invention to provide a compact solution for scanning three-dimensional object areas while minimizing the distortions due to parallax and other angle dependencies of the scene. The object is obtained with a unit and a system as specified in the accompanying claims.